Developing phoneme‐based lip‐reading sentences system for silent speech recognition

Lip-reading is a process of interpreting speech by visually analysing lip movements. Recent research in this area has shifted from simple word recognition to lip-reading sentences the wild. This paper attempts use phonemes as classification schema for explore an alternative and enhance system performance. Different schemas have been investigated, including character-based visemes-based schemas. The visual front-end model consists Spatial-Temporal (3D) convolution followed 2D ResNet. Transformers utilise multi-headed attention phoneme models. For language model, Recurrent Neural Network used. performance proposed testified with BBC Lip Reading Sentences 2 (LRS2) benchmark dataset. Compared state-of-the-art approaches sentences, demonstrated improved 10% lower error rate on average under varying illumination ratios.